Human Perception of Statistical Charts: An Introduction to Graphical Testing Methods

Pepperdine University

Emily Robinson

Cal Poly - San Luis Obispo

Reka Howard

University of Nebraska - Lincoln

Susan VanderPlas

University of Nebraska - Lincoln

Outline

Motivation and Background

Comprehensive Graphical Testing

Research Goals

Perception through Lineups

Prediction through ‘You Draw It’

Numerical Translation and Estimation

Overall Conclusions and Discussion

Introduction to Graphics

Data visualization is defined as the art of drawing graphical charts in order to display data Unwin (2020).

What are graphics useful for? Lewandowsky and Spence (1989)

📉 Data cleaning.

🔍 Exploring data structure.

💬 Communicating information.

Who uses graphics?

Governments (Harms 1991; Playfair 1801; Walker 2013).
Companies (Chandar, Collier, and Miranti 2012; Yates 1985).
News sources and mass media (Aisch et al. 2016).
Scientific publications (Gouretski and Koltermann 2007).

To get started, we are first going to lay the foundation of graphics. Data visualization has become central tool in modern data science and statistics. Unwin 2020 defines data visualization as the art of drawing graphical charts in order to display data.

Graphics are useful for data cleaning, exploring data structure, and communicating information.

In the 18th and 19th century, governments began using graphics to understand population and economic interests. In the 20th century, we saw companies using graphics to understand the inner workings of their business and support their business decision. We often see news source displaying graphics of weather forecasts such as hurricane trajectories. Today, we see graphics everywhere from scientific journals to mass media in the newspapers, TV, and internet.

Despite the popularity of graphics, we are too accepting of them as default without asking critical questions about the graphics we create or view (Unwin, 2020). We must begin asking ourselves How effective is this graph at communicating useful information?

Higher quality of technology has influenced the creation, replication, and complexity of graphics. We now have an infinitely many number design choices: + variables displayed, type of graphic, size of graphic, aspect ratio, colors, symbols, scales, limits, ordering of categorical variables

There is a need for an established set of concepts and terminology to build their graphics from so they can actively choose which of many possible graphics to draw in order to ensure their charts are effective at communicating the intended result.

History of Graphics

Visit the Timeline of Infographics by RJ Andrews (Info We Trust Data Storyteller).

Grammar of Graphics (Wilkinson 2012)

Graphics are viewed as a mapping from variables in the data set to visual attributes on the chart.

Building a masterpiece, by Allison Horst

Graphics are viewed as a mapping from variables in the data set to visual attributes on the chart.

aesthetics: links between data variables and graphical features (position, color, shape, size)
layers: geometric elements (points, lines, rectangles, text, …)
transformations: transformations specify a functional link between the data and the displayed information (identity, count, bins, density, regression). Transformations act on the variables.
scales: scales map values in data space to values in the aesthetic space. Scales change the coordinate space of an aesthetic, but don’t change the underlying value (so the change is at the visual level, not the mathematical level).
coordinate system: e.g.polar or Cartesian
faceting: facets allow you to split plots by other variables to produce many sub-plots.
theme: formatting items, such as background color, fonts, margins…

Software, such as Hadley Wickham’s ggplot2, aims to implement the framework of creating charts as the Grammar of Graphics recommends.

Testing Statistical Graphics

Evaluate design choices and understand cognitive biases through the use of visual tests.

Could ask participants to:

📉 identify differences in graphs.

📖 read information off of a chart accurately.

🌍 use data to make correct real-world decisions.

✏️ predict the next few observations.

Task complexity

Carpenter and Shah (1988) identifies pattern recognition, interpretative processes, and integrative processes as strategies and processes required to complete tasks of varying degrees of complexity.

Pattern recognition requires the viewer to encode graphic patterns.
Interpretive processes operate on those patterns to construct meaning.
Integrative processes then relate the meanings to the contextual scenario as inferred from labels and titles.

Lineup Protocal (Buja et al. 2009)

Introduction to Visual Inference

When doing exploratory data analysis, how do we know if what we see is actually there?

Lineup Protocol (Buja et. al, 2009)

Embed a target plot (actual data) in a lineup of null plots (randomly permuted data sets).

Introduction to Visual Inference

The main idea behind visual inference is that graphs are visual statistics or summaries of the data sets generated by mathematical functions. In a standard statistical analysis, a test statistic is generated from the data set and compared to the null distribution of that test statistic. Similarly, the visual statistic (target plot) is compared by a human viewer to other plots generated under the assumption of the null.

The main benefit of using visual inference is that visual tests tend to be more comprehensive. Since individuals are being asked to select one or more plots from a the lineup which are different but the difference is left unspecified the individual eye is actually spotting many differences at once. An equivalent numerical assessment may involved multiple tests using different test statistics.

Lineup Studies 📉 📊 📈

“You Draw It” (Robinson, Howard, and VanderPlas 2023)

“You Draw It” for News Media

New York Times (Aisch, Cox, and Quealy 2015)

“You Draw It” for Graphical Testing

Eye Fitting Straight Lines in the Modern Era (Robinson, Howard, and VanderPlas 2022)

youdrawitR Package

Perception of Logarithmic Scales

Exponential Growth

Von Bergmann (2021)

The motivation for this work came largly from the impact graphics and charts have had during the COVID-19 pandemic. Many of these graphics helped guide decision makers to implement policies such as shut-downs or mandated mask wearing, as well as facilitated communication with the public to increase compliance.

As graphics began to play an important role in sharing information with the public, it was crucial to start asking what made some of these charts better than others? Creators of the graphics were faced with many design choices in order to ensure their charts were effective at accurately communicating the current status of the pandemic.

A major issue we encountered in the creation of COVID-19 plots was how to display data from a wide range of values.

Problem: Data which spans several orders of magnitude shown on its original scale compresses the smaller magnitudes into relatively little area.
Solution: Use of a log scale transformation; alters the contextual appearance of the data.

However, as shown here, there is a general misinterpretation of exponential growth. This visual illustrates how individuals in public health may interpret exponential growth distinctly different from scientists during early, middle, and late stages of growth.

As Susan indicated, in order to make educated decisions when designing a chart, we need to establish guidelines through experimentation in order to ensure the graphic is effective at communicating the intended results.

Benefits and pitfalls of log scales

Burn-Murdoch et al. (2020)

Benefits were seen in spring 2020, during the early stages of the COVID-19 pandemic.

Large magnitude discrepancies in case counts at a given time point between different geographic regions.
Log scale transformations were usefulness for showing case count curves for areas with few cases and areas with many cases within one chart.

Pitfalls were exposed as the pandemic evolved, and the case counts were no longer spreading exponentially.

Graphs with linear scales seemed more effective at spotting early increases in case counts that signaled more localized outbreaks.
The effect of the linear scale appears to evoke a stronger reaction from the public than the log scale.

While this is one such example, there is a long history of using log scales to display results in ecology, psychophysics, engineering, and physics. Given the widespread use of logarithmic scales, it is important to understand the implications of their use in order to provide guidelines for best use.

Research objectives

Big Idea: Are there benefits to displaying exponentially increasing data on a log scale rather than a linear scale?

Perception through Lineups – tests an individual’s ability to perceptually differentiate exponentially increasing data with differing rates of change on both the linear and log scale.
Prediction with ‘You Draw It’ – tests an individual’s ability to make predictions for exponentially increasing data.
Estimation by Numerical Translation – tests an individual’s ability to translate a graph of exponentially increasing data into real value quantities.

The series of graphical tests were conducted through an RShiny application found at https://emily-robinson.shinyapps.io/perception-of-statistical-graphics-log/.

In this research, we conducted a series of three graphical studies to evaluate the impact displaying data on the log scale has on human perception of exponentially increasing trends compared to displaying data on the linear scale. Each study was related to a different graphical task, each requiring a different level of interaction and cognitive use of the data being presented.

The first experiment evaluated whether our ability to perceptually notice differences in exponentially increasing trends is impacted by the choice of scale. These perceptual differences are identified through pattern recognition.
The second study tested an individuals ability to make predictions for exponentially increasing data and required integrative processes by operate on the patterns to construct meaning.
The last and third study tested an individual’s ability to translate a graph of exponentially increasing data into real value quantitites and extend these estmiates by making comparisons.

Combined, the three studies provide a comprehensive evaluation of the impact of displaying exponentially increasing data on a log scale as it relates to perception, prediction, and estimation. The results of these studies help us make recommendations and provide guidelines for the use of log scales.

Study Participant Prompt: Which plot is most different?

Study Participant Prompt: Use your mouse to fill in the trend in the yellow box region.

Study Participant Prompt: From 4520 to 4540, the population increases by ____ Tribbles [Ewoks].

We used statistical lineups to test participants perception of exponentially increasing data.

Here we see an example of statistical lineups as shown in the first study. Notice how it is much easier to pick out panel 13 as being most different when displayed on the log scale (right) than on the linear scale (left).

In 2015, the New York Times introduced an interactive feature, called ‘You Draw It’, where readers input their own assumptions about various metrics and compare how these assumptions relate to reality. The Times team utilizes Data Driven Documents (D3) that allows readers to predict these metrics through the use of drawing a line on their computer screen with their mouse. We adapted this feature as a way to measure the patterns we see in data and compare intuitive visual model fitting and statistical model fitting. In the study, participants were shown a series of ‘You Draw It’ interactive task plots and asked to “Use your mouse to fill in the trend in the yellow box region.” The yellow box region moves along as the participant draws their trend-line until the yellow region disappears, providing a visual cue for the task.

Study Results

Perception
Prediction
Estimation

Each lineup plot evaluated was assigned a value based on the participant response (correct = 1, not correct = 0). The binary response was analyzed using generalized linear mixed model following a binomial distribution with a logit link function. Here we included both main effects and interactions for the curvature combination and scale while account for the participant and data specific characteristics with random effects.

Estimated (log) odds ratio of successfully identifying the target panel on the log scale compared to the linear scale. The y-axis indicates the the model parameters used to simulate the null plots with the target plot model parameter selection designated by shape and shade of green. The thumbnail figures on the right display the curvature combination as shown previously on both scales (linear - left, log - right).

The results indicated the choice of scale changes the contextual appearance of the data leading to slight perceptual advantages for both scales depending on the curvatures of the trend lines being compared.

Out of the study, we save the participant feedback data to a data base for analysis. Here we see a spaghetti plots of results. Participants drawn lines on the linear scale are shown in blue and the log scale are shown in orange. Variability in the statistically fitted regression lines occurred due to a unique data set being simulated for each individual; the gray band shows the range fitted values from the statistically fitted regression lines. Here we can see a strong underestimation on the linear scale (blue lines).

Allowing for flexibility, the we fit a Generalized Additive Mixed Model to estimate smoothing splines for the vertical residuals from the participant drawn line in relation to the NLS fitted values while accounting for participant variability.

The results from the analysis showed a clear underestimation of forecasting trends with high exponential growth rates when participants were asked to make predictions on the linear scale.

Conclusions

1. Perception through Lineups

Perceptual differences result from the contextual appearance (depends on choice of scale) of the trends.

2. Prediction through ‘You Draw It’

Clear underestimation of forecasting trends with high exponential growth rates when participants were asked to make predictions on the linear scale.

3. Numerical Translation and Estimation

Log logic is difficult and that we often misinterpret and miscalculate multiplicative reasoning.
Estimation accuracy for small magnitudes was improved by the use of the log scale, but sacrifices in accuracy on the log scale became apparent as magnitudes increased leading to advantages on the linear scale.

This research evaluated the use of log scales to display exponentially increasing data from three different angles and levels of complexity: perception, prediction, and estimation. Each study provided us insight into the advantages and disadvantages of displaying exponentially increasing data on a log scale and in what context each choice of scale might be more appropriate.

The first study laid the foundation for the future studies. Results from this study indicated the choice of scale changes the contextual appearance of the data leading to slight perceptual advantages for both scales depending on the curvatures of the trend lines being compared.

The second study tested participant’s abilities to make forecast predictions for exponentially increasing trends on both scales. The results from the analysis showed a clear underestimation of forecasting trends with high exponential growth rates when participants were asked to make predictions on the linear scale and improvements in forecasts were made when participants were asked to make predictions on the log scale.

The third study evaluated graph comprehension as it relates to the contextual scenario of the data shown. Overall, our results suggested that log logic is difficult and that anchoring and rounding biases result in a sacrifice in accuracy in estimates made on the log scale for large magnitudes.

Overall Recommendations

Perceptual advantages of the use of log scales due to the change in contextual appearance.
Our understanding of log logic is flawed when translating the information into context.
We recommend consideration of both user needs and graph specific tasks when presenting data on the log scale.
Caution should be taken when interpretation of large magnitudes is required, but advantages may appear when it is necessary to visually identify and interpret small magnitudes on the chart.

The results are instrumental in establishing guidelines for making design choices about scale which result in data visualizations effective at communicating the intended results.

We recommend understanding the contextual appearance log scales have on the data being displayed and the perceptual advantages and disadvantages that come with each. It is also important to recognize that our understanding of log logic is flawed when translating this information into context.

In general, it is important to consideration of both user needs and graph specific tasks when presenting data on the log scale. Caution should be taken when interpretation of large magnitudes is required, but advantages may appear when it is necessary to visually identify and interpret small magnitudes on the chart.

References

Aisch, Gregor, Nate Cohn, Amanda Cox, Josh Katz, Adam Pearce, and Kevin Quealy. 2016. “Live Presidential Forecast.” The New York Times. The New York Times. https://www.nytimes.com/elections/2016/forecast/president.

Aisch, Gregor, Amanda Cox, and Kevin Quealy. 2015. “You Draw It: How Family Income Predicts Children’s College Chances.” The New York Times. The New York Times. https://www.nytimes.com/interactive/2015/05/28/upshot/you-draw-it-how-family-income-affects-childrens-college-chances.html.

Buja, Andreas, Dianne Cook, Heike Hofmann, Michael Lawrence, Eun-Kyung Lee, Deborah F Swayne, and Hadley Wickham. 2009. “Statistical Inference for Exploratory Data Analysis and Model Diagnostics.” Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences 367 (1906): 4361–83.

Burn-Murdoch, John, Caroline Nevitt, Cale Tilford, Andrew Rininsland, Joanna Kao, Oliver Elliott, Emma Lewis, Brooke Fox, and Martin Stabe. 2020. “Coronavirus Tracked: Has the Epidemic Peaked Near You?” Coronavirus Chart: See How Your Country Compares. Financial Times. https://ig.ft.com/coronavirus-chart/?areas=eur.

Carpenter, Patricia A, and Priti Shah. 1988. “A Model of the Perceptual and Conceptual Processes in Graph Comprehension,” 26.

Chandar, Nandini, Deirdre Collier, and Paul Miranti. 2012. “Graph Standardization and Management Accounting at AT&t During the 1920s.” Accounting History 17 (1): 35–62.

Gouretski, Viktor, and Klaus Peter Koltermann. 2007. “How Much Is the Ocean Really Warming?” Geophysical Research Letters 34 (1).

Harms, Hans. 1991. “August Friedrich Wilhelm Crome (1753-1833) Autor Begehrter Wirtschaftskarten.” Cartographica Helvetica 3: 33–38.

Lewandowsky, Stephan, and Ian Spence. 1989. “The Perception of Statistical Graphs.” Sociological Methods & Research 18 (2-3): 200–242. https://doi.org/10.1177/0049124189018002002.

Loy, Adam, Lendie Follett, and Heike Hofmann. 2016. “Variations of q–q Plots: The Power of Our Eyes!” The American Statistician 70 (2): 202–14.

Majumder, Mahbubul, Heike Hofmann, and Dianne Cook. 2013. “Validation of Visual Statistical Inference, Applied to Linear Models.” Journal of the American Statistical Association 108 (503): 942–56.

———. 2014. “Human Factors Influencing Visual Statistical Inference.” arXiv Preprint arXiv:1408.1974.

Playfair, W. 1801. “The Statistical Breviary; Shewing, on a Principle Entirely New, the Resources of Every State and Kingdom in Europe, Wallis, Londres.” Press, Chicago.

Robinson, Emily A, Reka Howard, and Susan VanderPlas. 2022. “Eye Fitting Straight Lines in the Modern Era.” Journal of Computational and Graphical Statistics, 1–8.

———. 2023. “‘You Draw It’: Implementation of Visually Fitted Trends with R2d3.” Journal of Data Science 21 (2): 281–94.

Unwin, Anthony. 2020. “Why Is Data Visualization Important? What Is Important in Data Visualization?” Harvard Data Science Review, January. https://doi.org/10.1162/99608f92.8ae4d525.

VanderPlas, Susan, and Heike Hofmann. 2015. “Spatial Reasoning and Data Displays.” IEEE Transactions on Visualization and Computer Graphics 22 (1): 459–68.

———. 2017. “Clusters Beat Trend!? Testing Feature Hierarchy in Statistical Graphics.” Journal of Computational and Graphical Statistics 26 (2): 231–42.

VanderPlas, Susan, Christian Röttger, Dianne Cook, and Heike Hofmann. 2021. “Statistical Significance Calculations for Scenarios in Visual Inference.” Stat 10 (1): e337.

Von Bergmann, Jens. 2021. “Xkcd_exponential: Public Health Vs Scientists.” mountainMath. GitHub. March 17, 2021. https://github.com/mountainMath/xkcd_exponential.

Walker, Francis Amasa. 2013. “Statistical Atlas of the United States Based on the Results of the Ninth Census 1870 with Contributions from Many Eminent Men of Science and Several Departments of the Government.” [New York] J. Bien, lith.

Wilkinson, Leland. 2012. The Grammar of Graphics. Springer.

Yates, JoAnne. 1985. “Graphs as a Managerial Tool: A Case Study of Du Pont’s Use of Graphs in the Early Twentieth Century.” The Journal of Business Communication (1973) 22 (1): 5–33.

Human Perception of Statistical Charts: An Introduction to Graphical Testing Methods

Outline

Introduction to Graphics

History of Graphics

Grammar of Graphics (Wilkinson 2012)

Testing Statistical Graphics

Task complexity

Lineup Protocal (Buja et al. 2009)

Introduction to Visual Inference

Lineup Protocol (Buja et. al, 2009)

Introduction to Visual Inference

Lineup Studies 📉 📊 📈

“You Draw It” (Robinson, Howard, and VanderPlas 2023)

“You Draw It” for News Media

“You Draw It” for Graphical Testing

Perception of Logarithmic Scales

Exponential Growth

Benefits and pitfalls of log scales

Research objectives

Graphical Tasks

Study Results

Conclusions

Overall Recommendations

References

Emily A. Robinson

erobin17@calpoly.edu

github.com/earobinson95